On-line Reinforcement Learning Using Incremental Kernel-Based Stochastic Factorization SUPPLEMENTARY MATERIAL

This is the supplementary material for the paper entitled “On-line Reinforcement Learning Using Incremental Kernel-Based Stochastic Factorization” [2]. It contains the details of our theoretical developments that could not be included in the paper due to space constraints. This material should be read in conjunction with the main paper. 1 Preliminaries • Similarly to Ormoneit and Sen [3], we define a “mother kernel” φ(x) : R+ 7→R+ satisfying (i) φ(x) is continuous in R+, (ii) ∫ ∞ 0 φ(x)dx≤ Lφ < ∞, (iii) φ(x)≥ φ(y) if x < y, (iv) ∃ Aφ ,λφ > 0,∃ Bφ ≥ 0 such that Aφ exp(−x)≤ φ(x)≤ λφ Aφ exp(−x) if x≥ Bφ . Remarks: – Assumption (i) is implied by Ormoneit and Sen’s [3] assumption that φ is Lipschitz continuous. Ormoneit and Sen also assume that ∫ 1 0 φ(z)dz = 1 (see Appendix A.1 in [3]). – Assumption (iv) implies that the kernel function φ will eventually decay exponentially and also that φ(z)> 0 for all z ∈ R+. • Let S⊂ [0,1]d and let ‖ · ‖ be a norm in Rd . Then, we define kτ(s,s) = φ ( ‖ s− s′ ‖ τ ) , where τ > 0 is the “width” of the kernel kτ . • Let M be a Markov decision process (MDP) with state space S and let Sa = {(sk ,ra k , ŝk)|k = 1,2, ...,na} be a set of sample transitions associated with action a ∈ A, where sk , ŝk ∈ S and ra k ∈ R. We define the normalized kernel function associated with action a as κa τ (s,s a i ) = kτ(s,si ) ∑a j=1 kτ(s,s a j) . • Let S̄≡ {s̄1, s̄2, ..., s̄m} be a set of representative states in S. Define: – ŝ∗ ≡ ŝk with k = argmaxi min j ‖ ŝi − s̄ j ‖,